Versions Compared

Old Version 9

changes.mady.by.user Sarah Bigault

Saved on

compared with

New Version Current

changes.mady.by.user Borja Moro Moreno

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Expand
titleWatch video tutorial
Widget Connector
overlayyoutube
_templatecom/atlassian/confluence/extra/widgetconnector/templates/youtube.vm
width600px
urlhttps://www.youtube.com/watch?v=Av7lLjK2Mw8
height300px

...

While Devo is designed to receive real-time data feeds, it's sometimes useful to upload static files if you want historical log data to be accessible in Devo for querying and analysis. These static files will always be stored in tables starting with the my.upload tag levels. If you need to send event files to a table other than a my.upload, please contact us.

What permissions do I need?

To be able to upload files, you need to have the Manage version of the Data Uploadupload permission. If you do not have this permission, you will not be able to see the option in the navigation pane and thus you will not be able to access this area to upload files (know more about permissions here).

...

Upload a data file

You have the option of uploading Follow these steps to upload a file from your local machine or from a Dropbox account. to your Devo domain.

Note

Upload limits

  • File size → Note that there is a 2GB limit on files uploaded from the local machine

...

Note

  • .

  • Event length → Note that there is a limit

per event of 10000 bytes in the file uploaded using this method

  • of 10KB per line (equivalent to 10,000 characters).

Rw ui steps macro
Rw step

Once inside the Data upload area, select either Dropbox or Local files. 

  • If you select Dropbox, you will be prompted to grant Devo access to your Dropbox files, then log into Dropbox. Navigate to the folder and select the file you want to upload. 

    • If you select Local files,

    you can either drag and drop the file or navigate to the necessary folder and select it for upload. You can try the process using this file:

    View file
    namemarkets.sample

    Rw step

    When you have selected the file, click the Upload FileSettings button. The settings page appears.

    Image RemovedImage Added

    This page contains a couple of fields that will determine how the data will be saved in Devo.

    Tag

    The first two levels of the tag are predefined as my.upload. The third and fourth levels are up to you. Try to name them in a way that will help you identify the data contained in the data table.

    Date parsing type

    The date assigned to the events when parsing the data contained in your file. You have two options: can either use the date at the moment of uploading the file or use a date contained in the event information.

    • If you choose Current date, each event will be assigned the date at the moment of uploading the file.

    • If you choose Date from a log field, each event will be assigned a date that is included in their information. You must then indicate the Parsing log type so that Devo can scan the events and highlight the fields that it recognizes as possible date/time information. You can select one of the following options as a parsing type:

      1. CSV: the file is treated as a typical CSV in which the sequence of fields is constructed using commas as delimiters. The resultant fields appear below for you to select the desired one to extract the date. 

      2. Delimiter: if the delimiter in your file is not a comma, you can specify a different one. You just need to type the delimiter that matches the one contained in your file and then select one of the fields below to extract to date.

      3. Syslog: if your file complies with the syslog structure, you can select this option and the fields with a valid date format will appear below for you to select.

      4. Regular expressionExpression: you can use a regular expression to establish a search pattern to find a piece of information that expresses a date. If you want to know more about regular expressions, the language they use and the syntax to construct them you can click the following link.

    Note

    The fields that appear in light grey cannot be selected because they do not match the accepted format while the fields in green are the ones that coincide with a format recognized by Devo. Check the table at the end of the article to see the accepted date formats as well as some of the most recurrent regular expressions to search for dates.

    Image RemovedImage Added
    Rw step

    Click the Confirm Settings Summary button. 

    Note

    When uploading a CSV fileIn some cases, you may experience a browser crash due to conflicts with external cookies. To avoid this issue, clear your browser’s cookies and reload your browser before uploading the file.

    ...

    To work with the new data, go to Data search and use the finder to locate the my.upload table that you just created. This table will always be created with the following three columns:

    eventdate

    This is the date in which the data arrives into Devo or the one specified using a log field.

    uploadid

    This is the coding that identifies the data according to the moment it is uploaded. Each new upload will have a unique uploadid.

    message

    Use the Split (split) function to manually parse the message of the column.

    ...

    Note

    Working with JSON files?

    Please bear in mind that if you are uploading a file in JSON format, it will show in Data Search as a string data type in the search window. To parse this string to JSON data type, use the to json operation.

    ...

    Devo is capable of recognizing a wide range of date formats in the files you upload and you can use regular expressions to find them and use them.

    Format

    Sample

    Regular expression

    DD/MM/YYYY

    30/05/2017

    \d{2}\/\d{2}\/\d{4}

    DD/MM/YYYY HH:mm:ss

    30/05/2017 18:00:00

    \d{2}\/\d{2}\/\d{4}\s\d{2}\:\d{2}\:\d{2}

    DD:MM:YYYY

    30:05:2017

    \d{2}\:\d{2}\:\d{4}

    DD:MM:YYYY-HH:mm:ss

    30:05:2017-18:00:00

    \d{2}\:\d{2}\:\d{4}\-\d{2}\:\d{2}\:\d{2}

    ddd MMM DD HH:mm:ss YYYY

    Tue May 30 18:00:00 2017

    \w{3}\s\w{3}\s\d{2}\s\d{2}\:\d{2}\:\d{2}\s\d{4}

    DD-MMM-YYYY HH:mm:ss.SSS

    30-May-2017 18:00:00.123

    \d{2}\-\w{3}\-\d{4}\s\d{2}\:\d{2}\:\d{2}\.\d{3}

    DD-MM-YYYY

    30-05-2017

    \d{2}\-\d{2}\-\d{4}

    MMM dd HH:mm:ss

    May 30 18:00:00

    \w{3}\s\d{2}\s\d{2}\:\d{2}\:\d{2}

    x (epoch milliseconds with dot separator)

    1496167200

    \d{10}\.\d{3}

    YYYY/MM/DD

    2017/05/30

    \d{4}\/\d{2}\/\d{2}

    YYYY/MM/DD HH:mm:ss

    2017/05/30 18:00:00

    \d{4}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}

    YYYY/MM/DD hh:mm:ss

    2017/09/14 15:10:23

    \d{4}\/\d{2}\/\d{2}\s\d{2}\:\d{2}\:\d{2}

    YYYY:MM:DD

    2017:05:30

    \d{4}\:\d{2}\:\d{2}

    YYYY:MM:DD-HH:mm:ss

    2017:05:30-18:00:00

    \d{4}\:\d{2}\:\d{2}\-\d{2}\:\d{2}\:\d{2}

    YYYY-MM-DD

    2017-05-30

    \d{4}\-\d{2}\-\d{2}

    YYYY-MM-DD hh:mm:ss

    2017-09-14 15:10:23

    \d{4}\-\d{2}\-\d{2}\ s\d{2}\:\d{2}\:\d{2}

    YYYY-MM-DD HH:mm:ss.SSS

    2017-05-30 18:00:00.123

    \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}

    YYYY-MM-DD HH:mm:ss.SSSSSS

    2013-01-17 00:54:01.744909

    \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{6}

    YYYY-MM-DDThh:mm:ss

    2017-09-13T19:01:15

    \d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}

    YYYY-MM-DDThh:mm:ss.SSSZ

    2017-09-13T19:01:15.000Z

    \d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}\.\d{3}Z

    YYYY-MM-DDThh:mm:ssZ

    2017-09-13T19:01:15Z

    \d{4}-\d{2}-\d{2}T\d{2}\:\d{2}\:\d{2}Z