Uploading log files

[ 1 What permissions do I need? ] [ 2 Upload a data file ] [ 3 Accessing tables with uploaded data ] [ 4 Recognized date formats ]

While Devo is designed to receive real-time data feeds, it's sometimes useful to upload static files if you want historical log data to be accessible in Devo for querying and analysis. These static files will always be stored in tables starting with the my.upload tag levels. If you need to send event files to a table other than a my.upload, please contact us.

What permissions do I need?

To be able to upload files, you need to have the Manage version of the Data Upload permission. If you do not have this permission, you will not be able to see the option in the navigation pane and thus you will not be able to access this area to upload files.

Upload a data file

You have the option of uploading a file from your local machine or from a Dropbox account. Note that there is a 2GB limit on files uploaded from the local machine so if you have an extremely large file, you should first upload it to Dropbox.

Note that there is a limit per event of 10000 bytes in the file uploaded using this method.

Accessing tables with uploaded data

To work with the new data, go to Data search and use the finder to locate the my.upload table that you just created. This table will always be created with the following three columns:

eventdate	This is the date in which the data arrives into Devo or the one specified using a log field.
uploadid	This is the coding that identifies the data according to the moment it is uploaded. Each new upload will have a unique uploadid.
message	Use the Split (split) function to manually parse the message of the column.

Working with JSON files?

Please bear in mind that if you are uploading a file in JSON format, it will show in Data Search as string data type. To parse this string to JSON data type, use the to json operation.

Recognized date formats

Devo is capable of recognizing a wide range of date formats in the files you upload and you can use regular expressions to find them and use them.

Format	Sample	Regular expression

Format	Sample	Regular expression
DD/MM/YYYY	30/05/2017	`\d{2}\/\d{2}\/\d{4}`
DD/MM/YYYY HH:mm:ss	30/05/2017 18:00:00	`\d{2}\/\d{2}\/\d{4}\s\d{2}\:\d{2}\:\d{2}`
DD:MM:YYYY	30:05:2017	`\d{2}\:\d{2}\:\d{4}`
DD:MM:YYYY-HH:mm:ss	30:05:2017-18:00:00	`\d{2}\:\d{2}\:\d{4}\-\d{2}\:\d{2}\:\d{2}`
ddd MMM DD HH:mm:ss YYYY	Tue May 30 18:00:00 2017	`\w{3}\s\w{3}\s\d{2}\s\d{2}\:\d{2}\:\d{2}\s\d{4}`
DD-MMM-YYYY HH:mm:ss.SSS	30-May-2017 18:00:00.123	`\d{2}\-\w{3}\-\d{4}\s\d{2}\:\d{2}\:\d{2}\.\d{3}`
DD-MM-YYYY	30-05-2017	`\d{2}\-\d{2}\-\d{4}`
MMM dd HH:mm:ss	May 30 18:00:00	`\w{3}\s\d{2}\s\d{2}\:\d{2}\:\d{2}`
x (epoch milliseconds with dot separator)	1496167200	`\d{10}\.\d{3}`
YYYY/MM/DD	2017/05/30	`\d{4}\/\d{2}\/\d{2}`
YYYY/MM/DD HH:mm:ss	2017/05/30 18:00:00	`\d{4}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}`
YYYY/MM/DD hh:mm:ss	2017/09/14 15:10:23	`\d{4}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}`
YYYY:MM:DD	2017:05:30	`\d{4}\:\d{2}\:\d{2}`
YYYY:MM:DD-HH:mm:ss	2017:05:30-18:00:00	`\d{4}\:\d{2}\:\d{2}\-\d{2}\:\d{2}\:\d{2}`
YYYY-MM-DD	2017-05-30	`\d{4}\-\d{2}\-\d{2}`
YYYY-MM-DD hh:mm:ss	2017-09-14 15:10:23	`\d{4}\-\d{2}\-\d{2}\ s\d{2}\:\d{2}\:\d{2}`
YYYY-MM-DD HH:mm:ss.SSS	2017-05-30 18:00:00.123	`\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}`
YYYY-MM-DD HH:mm:ss.SSSSSS	2013-01-17 00:54:01.744909	`\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}`
YYYY-MM-DDThh:mm:ss	2017-09-13T19:01:15	`\d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}`
YYYY-MM-DDThh:mm:ss.SSSZ	2017-09-13T19:01:15.000Z	`\d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}\.\d{3}Z`
YYYY-MM-DDThh:mm:ssZ	2017-09-13T19:01:15Z	`\d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}Z`