Proprietary Data Upload
This page describes how to do proprietary uploads of data not using the Avail Data Exporter
Proprietary Upload
Upload to upload.avail.net.
Use FTP with explicit TLS/SSL to upload.
Log in to the ftp using the customer id and password provided to you by Avail.
File encoding: UTF-8 (with or without BOM)
Files: gzip:ed files. The files inside the gziped files can have any name.
Finish uploads with file done.txt that contains at least one character.
This file tells the server that you are done with your upload.
File Names
| Category Data |
categorydata.gz |
| Category Names |
categorynamesdata.gz |
| Valid Products |
validdata.gz |
| Product Data |
productdata.gz |
| Transactions |
transactiondata.gz |
| Upload done marker |
done.txt |
Delimiters
| Text Qualifier |
|z6CEYek4| |
| Row Delimiter |
Unix line-break (\n) |
| Column Delimiter |
; |
Enclose all values within text qualifiers.
The reason for this unorthodox qualifier is to enable easy export from databases such as MS SQL that do not support escaping.
Empty strings will assume value null for int, float and date. For string empty string is an empty string.
Example Data Rows
|z6CEYek4|1234|z6CEYek4|;|z6CEYek4|Some "quoted" text|z6CEYek4|;|z6CEYek4|Field containing new line
and tabs|z6CEYek4|
|z6CEYek4|1235|z6CEYek4|;|z6CEYek4|Another field|z6CEYek4|;|z6CEYek4|Third column, second row|z6CEYek4|
Resources
Below are some help and tools.
There is no need to use the below, this is only for those that not already have a preferred way of accomplishing the data export and upload.
Sample Export from MySQL
SELECT * INTO OUTFILE 'c:/transactiondata.txt'
FIELDS TERMINATED BY ';' ENCLOSED BY '|z6CEYek4|'
LINES TERMINATED BY '\n'
FROM orders;
File Encoding Conversion on Linux, Unix and Mac
The code below converts a file from ISO-8859-1 (a.k.a. Latin-1) to UTF-8.
iconv -f ISO_8859-1 -t UTF-8 transactiondata.txt > transactiondata-utf-8.txt
Gzip:ing files on Windows
Windows have not native support for gzip.
7-zip is an open source project that has a command line interface.
Use below syntax to create a gzip file for category data.
7z a categorydata.gz MyCategoriesFile.txt
Uploading Files on Windows
WinSCP is an open source project that has a command line interface.
Download version 4.2.1 or greater.
You need to set up a connection using the graphical interface.
Make sure Advanced Options is checked.
Set Host name to "upload.avail.net" and enter your user name and password
Set File Protocol to "FTP" and "TLS Explicit encryption"
Under Connection check Passive
Save and enter a name for example "MyAvailTransfer" and check Save password
Use below syntax to upload a set of gz-files and finishing with a done.txt file.
winscp MyAvailTransfer /command "option transfer binary" "put *.gz" "put done.txt" "exit"
Uploading Files on Linux, Unix and Mac
Below is an example of how data can be uploaded using cURL.
curl -u xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:MyPassword --ftp-ssl-reqd -3 -T "{transactiondata.gz,productdata.gz,done.txt}" ftp://upload.avail.net/
Manual Uploads
To manually upload files - i.e. no command line interface for scripting uploads - the
FileZilla client can be used. Runs on Windows, Mac and Linux.
Use settings:
Host: upload.avail.net
Servertype: FTPES - FTP over explicit TLS/SSL
Logontype: Normal
User: CustomerId
Password: Password