What is it?
The Data Gateway is a tool that offers a way of reaching multiple on-premise resources from a TARGIT Cloud Solution.
The Data Gateway does this without requiring users to open ports in their firewalls.
The Data Gateway serves multiple purposes.
Some of the tasks the Data Gateway covers can also be covered in other ways, while other jobs require using the Data Gateway.
- You can use the Data Gateway to connect to On-Premise data sources.
-
Data Sources that can be connected through TARGIT Management and used directly for reporting
-
Data Sources that can be connected through ETL Studio and used in one or more ETL jobs
In both cases, you set up the connection on-premise using the Data Gateway, and then you can refer to that connection when utilizing the data in TARGIT Cloud.
This task can also be done by adjusting the firewall settings on the server where your on-premise data sources are located.
This is an alternative to using the Data Gateway for this purpose, described in detail here.
Note:
Directing queries through an additional component (the Data Gateway) always incurs an overhead.
Through testing, we have seen no noticeable overhead on smaller queries and up to 40% overhead on larger queries.
-
Data Sources that can be connected through TARGIT Management and used directly for reporting
- You can use the Data Gateway to set up export folders that can be utilized in scheduled jobs in a TARGIT Cloud solution that delivers PDFs, snapshots, and excel/CSV files in an on-premise folder.
You can theoretically do this in other ways, but currently, this is the recommended way of setting up a connection to on-premise export folders in TARGIT Cloud.
- You can use the Data Gateway to set up on-premise folders that Data Discovery uses for data sources when using TARGIT Cloud.
You can theoretically do this in other ways, but currently, this is the recommended way of setting up on-premise folders used for Data Discovery in your TARGIT Cloud solution.
Where can I get it?
When you log on to your TARGIT Cloud Control Panel (see here for more info on this), you have three links at the bottom - one of them being Download:
By clicking this, you get access to (among other links) the gateway (Gateway Connections):
| Note: .NET 8 Runtime needs to be installed before installing the Gateway client. |
When clicking the Gateway connections, a download will automatically begin - once it is done, you can double-click this file to start the installation:
If you get this - click More info:
And Run Anyway:
Now agree to the license conditions and click Install:
Now the Gateway installs.
Important: If TARGIT Gateway Client is installed on a different server than the data warehouse/files, then make sure that the service “TARGIT Gateway Client” is running as a user with the necessary rights to access data and files.
After changing service account, make sure that the user used for the service has write permission in the folder C:\ProgramData\TARGIT\GatewayClientService. It is required for the service to make changes to the configuration.
How to use it
Connect to on-premise data sources
The Data Gateway runs as a service - just like the TARGIT server in an on-premise solution:
The account that runs this service (in the example above, Local System) needs read rights to every data source you wish to access through the Data Gateway.
Now start the Data Gateway with this short-cut:
The main dialogue looks like this:
The name field could be kept as the server name.
The Gateway Connection string needs to be copied from your Cloud Control Panel, which is how your Data Gateway connects to your cloud environment.
Example of a connection string from Cloud Control Panel:
Important note: Although the Gateway client will work with just having configured the 'Gateway connection string', we recommend that this is only used for testing. In Production environments we strongly recommend to use the 'Direct URL' setting as well - see "Configuring the Gateway client for Direct Access" below in this article. By using 'Direct URL' you will achieve gains in performance as well as stability.
The three tabs - databases, shared folders, and shared files represent the functionality covered by the Data Gateway.
1. tab Databases...
...is where you make a connection to your on-premise databases, relational or analytical databases. If needed, you can also install drivers for your database connections
Here you can connect a connection to a database -like this example:
Now you can use TARGIT Management - and refer to the connection mentioned above - connecting through the Data Gateway:
This connection can also be utilized from ETL Studio - when you create a Data Source; you have a field called Gateway connection string:
This is where you paste your connection string copied from your Cloud Control Panel. (see more here on the Cloud Control Pane.).
2. tab Shared Folders...
...is where you set up folders that can be used for export from scheduled jobs or/and used as input for your Data Discovery in the Cloud Environment.
Here you can add new local export folders meant to be used in your cloud implementation of TARGIT:
When you open TARGIT Management and pick the export folder tab, you can now add an entry that refers to your Data Gateway setup:
Now, this export folder is available in the Windows Client for scheduled export jobs.
3. tab Shared Files...
...is where you set up access to files that are used as input in the Data Discovery running in your Cloud Environment
Configuring the Gateway client for Direct Access (optional)
For performance reasons we recommend to to configure Direct Access to the server.
The screenshot above is based on this example:
- Prerequisites
- External DNS name e.g., extgateway.domain.com.
- SSL certificate for external DNS name. It can be a wildcard certificate like *.domain.com.
- Decide which port to use, no specific port is required. E.g., port 5000.
- Forward this port and the external DNS name/IP to the internal gateway server.
- Open the port in the gateway server local firewall.
- Open TARGIT Gateway Client Configuration tool.
- Enter direct URL and port, e.g., https://extgateway.domain.com:5000/
- Click “Save Changes”.
- In the folder C:\Program Files (x86)\TARGIT\TARGIT Gateway Client open appsettings.json. You can use Notepad, but make sure you open with admin privileges to be able to overwrite the file.
- Enter configuration depending on how certificate is stored. See sample settings appendix for further information regarding configuration. Remember that for file paths a backslash (\) should be written as (\\).
- Restart the TARGIT Gateway Client service.
Sample appsetting.json
Note: You can open this URL for information about configuration options, e.g., different types of certificate files: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel/endpoints
Sample 1: Certificate stored certificate store
The comments, /* ... */, in this example are for informational purposes only and should be removed , otherwise the JSON file will fail.
{
"Logging": {
"LogLevel": {
"Default": "Warning"
}
},
"AllowedHosts": "*",
"Kestrel": {
"Endpoints": {
/* Endpoint for ConfigurationTool access */
"ConfigurationTool": {
/* Do not change */
"Url": "http://localhost:8461"
},
/* Endpoint for data access */
"Https": {
/*
Listening url and port, the selected port must be open in the firewall for external access
*/
"Url": "https://*:5000",
"Certificate": {
"Subject": "<certificate subject name in store>",
/*
Use "My" for personal certificates
See this URL for more information about store names:
https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.x509certificates.storename
*/
"Store": "<store name>",
/* LocalMachine or CurrentUser */
"Location": "<store location>"
}
}
}
}
}
(Sample file, without comments, attached at the bottom of this article.)
Sample 2: Certificate stored in a certificate .pfx file
The comments, /* ... */, in this example are for informational purposes only and should be removed , otherwise the JSON file will fail.
{
"Logging": {
"LogLevel": {
"Default": "Warning"
}
},
"AllowedHosts": "*",
"Kestrel": {
"Endpoints": {
/* Endpoint for ConfigurationTool access */
"ConfigurationTool": {
/* Do not change */
"Url": "http://localhost:8461"
},
/* Endpoint for data access */
"Https": {
/*
Listening url and port, the selected port must be open in the firewall for external access
*/
"Url": "https://*:5000",
"Certificate": {
/*
This configuration expects the certificate to be exported as PKCS #12 format.
*/
"Path": "<Path to .pfx file>",
"Password": "<Password for .pfx file>"
}
}
}
}
}
(Sample file, without comments, attached at the bottom of this article.)
The technical stuff
The Gateway effectively creates an isolated area where the Data Gateway client and TARGIT Cloud/ETL Studio can communicate.
Note: This means that you won't have to open any of the ports mentioned below manually
The current version of TARGIT Data Gateway (as of September 2022) uses Azure Service Bus to communicate between various peers.
Azure Service Bus is a message broker that is available in Microsoft Azure.
When the Gateway Client service starts, it connects to the TARGIT Cloud Control panel API via HTTPS port 443 to get access to the Azure Service bus. It uses the key specified in the connection string to authenticate with the Cloud Control panel API. The Control panel API verifies that the key is correct and issues two temporary access tokens for the Azure Service Bus that can only access resources in your TARGIT Cloud tenant.
When the Data Gateway client receives the access tokens, it uses these to directly connect to the Azure Service Bus using AQMP protocol on ports 5671 and 5672. It will start receiving packets and respond to any incoming requests.
The response is divided into smaller chunks and sent as packets over Azure Service Bus.
Comments
What are the recommended system requirements for the server hosting the gateway?
Andrew Cason It is very hard to tell, since it depends very much on the queries being passed. It is mainly heavy on the memory consumption. Whether that is a few hundred megabytes or a few gigabytes depends on the throughput. It hasn't been a big issue for customers.
Jakob Bak, do you have a standard set of requirements? I need to provide this to our sys admins who are moving the gateway to a server that hosts many gateways.
Andrew Cason We don't carry a standard set of requirements. I can tell you that our own internal gateway takes up 850 MB with around 50 active users and 200+ scheduled jobs. It runs on a Xeon E5-2650 v2 @ 2,6 GHZ with 8 cores and 32 GB RAM.
Thank you, that is helpful. Your software is so efficient!
Thanks Andrew Cason - appreciate it.
Great overview, thanks for sharing Jakob Bak and Niels Thomsen!
Please sign in to leave a comment.