Influx with RequestHttpx

Dean_Nederveld · October 30, 2019, 6:03pm

MOVED TO STAFF NOTE (610.5 KB)
Please the request file included with this reply. Thank you for your assistance.

Tim_hms · October 30, 2019, 6:08pm

Hi Dean,

One thing I noticed on this was that it looks like your device doesn’t seem to have a very strong signal strength. The minimum signal strength we recommend for a strong connection is 18. Do we have a way to try and move the device/antenna to see if we can improve the signal and see if it’s still running into issues?

Dean_Nederveld · October 30, 2019, 6:36pm

This is device I use for development. It is configured to use Ethernet as the primary interface with cellular as the fallback.

Tim_hms · October 30, 2019, 6:39pm

Just to check was there a time when the data was successfully being written to the database initially or has there been no data written to it yet? If so could we try and slow down the timer and see if it is still running into issues?

Dean_Nederveld · October 30, 2019, 8:38pm

I have been able to repeatedly write a single string of data to the REST API of the database using the REQUESTHTTPX command with the POST method. The database uses the same method to ingest a text file containing several strings of data in the same format as the single string. I have not been able to ingest this file of data via the eWON.

I can copy the file from the eWON to my laptop and, using an app called Postman, I can write this file to the database. In Postman, to write a string of data to the REST API I choose the ‘raw’ option. To write the file of data I choose the ‘binary’ option. I don’t know what Postman does differently when it executes the command but I do know I have to manually enter in the body the single data string when choosing the ‘raw’ option and I am allowed to select the text file of the multiple strings when choosing the ‘binary’ option.

Tim_hms · October 30, 2019, 8:44pm

We do have a way to save the data as a binary files in your SD card if that would make it possible to post to Influx.

https://ewonsupport.biz/ebd/

Dean_Nederveld · October 31, 2019, 1:04pm

@Tim_hms, my script successfully creates and writes to a text file on the SD card, although it’s not using the EBD method. This is the section that writes a single string of data in the format required by the database:

   OPEN "file:" + $DataPostLoc$ FOR BINARY APPEND AS 1
   PUT 1, $DataToPost$ + CHR$(10)
   CLOSE 1

The file is successfully written to, and I can copy that file to my laptop and send it to the database via Postman.

What I cannot do is use the REQUESTHTTPX command to send the file to the database, although I believe I should be able to.

Dean_Nederveld · October 31, 2019, 1:05pm

@Tim_hms, I was able to successfully use the SENDMAIL command to email myself the file from the SD card as an attachment,

SENDMAIL “email@gmail.com”, “”, “TEST”, “&[$dtUF$ftT$uf” + $fileToSend$ + “]”

I note, however, that the attached file name is “$dtUF$ftT$uf/usr/sdext/datatopost.txt.txt”. I believe this may be the problem if the database sees all the extra characters other than the simple file name.

Tim_hms · October 31, 2019, 1:06pm

I believe you should be able to fix this by using the filename part of the EBD

so you could do something like this $fnyourfilename

Dean_Nederveld · October 31, 2019, 7:51pm

@Tim_hms, I agree with adding the $fn switch and was on that track this morning. I first tested it with the EBD helper and the results were as I expected. The filename used to save to my laptop with the $fn switch was the simple filename I expected.

I also tried using the SENDMAIL command but I had an issue with AWS Simple Email Service (SES) requiring a STARTTLS instruction, so I switched to relay.talk2m.com for testing and did prove receipt of an email with the data file attached with the simple file name ‘datatopost.txt’ without the extraneous EBD characters. Progress!

This worked –
SENDMAIL “email@gmail.com”, “”, “TEST”, “&[$dtUF$ftB$uf/usr/sdext/datatopost$fndatatopost.txt]”

However, the REQUESTHTTPX command, now with the $fn switch, still isn’t sending the POST to the database in a manner in which the database can ingest the data.

This still isn’t fully working –
REQUESTHTTPX url$, method$, header$, “”, “[$dtUF$ftB$uf/usr/sdext/datatopost$fndatatopost.txt]”

Dean_Nederveld · October 31, 2019, 7:51pm

I dumbed down the script so I could just force a single REQUESTHTTPX command each time I stop and start the script. It seems no matter what I place in the header, so long as it is not null and it starts with Content-Type=, the command executes and I receive the anticipated response from the database server. I can even leave the header as Content-Type= with nothing after the equal sign and it seems to execute but no data is written.

POST_DATA:
FileSize% = FS "size", "/usr/sdext/datatopost"
url$ = "http://IP_ADDRESS:PORT/write?db=import"
method$ = "POST"
//header$ = "Content-Type=text/plain;charset=utf-8"
header$ = "Content-Type=X"
REQUESTHTTPX url$, method$, header$, "", "&[$dtUF$ftB$uf/usr/sdext/datatopost$fndatatopost.bin]"
actionID% = GETSYS PRG,"ACTIONID"
PRINT "Request actionID: "; actionID%
ONSTATUS "GOTO ON_EVENT" 
END

ON_EVENT:
eventID% = GETSYS PRG,"EVTINFO"
IF (eventID% = actionID%) THEN
  SETSYS PRG,"ACTIONID",eventID%
  stat% = GETSYS PRG,"ACTIONSTAT"
  IF (stat% = 0) THEN
    GOTO RESPONSE
  ELSE
    PRINT "Error: " + STR$ stat%
  ENDIF
ENDIF
END

RESPONSE:
a$ = RESPONSEHTTPX "STATUSCODE"
$LogText$ = "***POST FILE ** STATUSCODE: " + a$ + " ACTIONSTAT: " + STR$ stat%
b$ = RESPONSEHTTPX "HEADER"
PRINT "***all headers: " + b$;
$FileSize% = FS "size", "/usr/sdext/datatopost"
$LogText$ = $LogText$ + " FILESIZE: " + STR$ $FileSize%
PRINT $LogText$
LOGEVENT $LogText$
END

Each time I execute the above, I receive this response from the database server:

When I send the same file via the Postman app I receive this response:

The one thing I notice as different between the responses is the response to the eWON includes connection: close whereas that is not include in the response to Postman.

Dean_Nederveld · October 31, 2019, 7:51pm

I changed the script a bit so the posting of the file of data would wait five (5) seconds after the posting of the string of data. I did this so I can compare the response from the database server from each REQUESTHTTPX command.

This is the response from the execution of the REQUESTHTTPX command to send a single string of data to the database server, and a successful writing or ingesting of the data into the database:

I have been able to execute this command successfully over and again.

This is the response from the execution of the REQUESTHTTPX command to send a text file of multiple strings of data to the database server, but the data is not successfully written or ingested into the database:

Note, the responses are virtually identical even though only the first one is successful.

Tim_hms · October 31, 2019, 7:59pm

Hi Dean,

I think I might need to escalate this to one of my colleagues in Belgium to get a better answer on why we might be running into this issue

Dean_Nederveld · October 31, 2019, 8:34pm

@Tim_hms, Thank you. I look forward to the comments from you and your colleague(s).

Tim_hms · October 31, 2019, 9:00pm

Can you send us what you’re doing in Postman?

Tim_hms · October 31, 2019, 9:03pm

On the server side of this can you tell us what response is given for an individual string of data vs when you send multiple strings of data?

Dean_Nederveld · November 1, 2019, 6:01pm

Below is the setup on the Postman Params tab:

Below is the setup on the Body tab:

Where datatopost is a text file on my laptop identical to the text file on my device that I am trying to send to my database.

Below are the header responses:

The response 204 No Content plus checking the database to see that the data was successfully written is my indication that the Postman command executed properly.

You asked “On the server side of this can you tell us what response is given for an individual string of data vs when you send multiple strings of data?” I will have to do some research on Influx to see if anything is noted or logged on the server side.

Dean_Nederveld · November 1, 2019, 6:01pm

@Tim_hms, I believe I was able to get some information from the server side logs for a particular REQUESTHTTPX command I executed. I modified my script so I could execute just one REQUESTHTTPX command to write a text file of data to the database and then stop. Here is the response from the database server as shown in the BASIC IDE console of my eWON:

This is the server side log entry for the above command:

[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:13:53:10 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "-" f1040bde-fcae-11e9-9bbe-0e5b008f42bc 81

The id’s and the timestamps are the same so that’s the information from the transaction. I do note that the file name is not in the log entry on the server side.

Below is the response from the database server as shown in the BASIC IDE console of my eWON of sending a single string of data (in the S4 Post-Data switch of the REQUESTHTTPX command):

This is the server side log entry for the above command:

[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:14:05:42 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "-" b141aae6-fcb0-11e9-b009-0e5b008f42bc 12457

Finally, here is the same information from write a text file of data to the database from Postman:

[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:14:11:20 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "PostmanRuntime/7.19.0" 7a856350-fcb1-11e9-ba77-0e5b008f42bc 334349

deryck_hms · November 1, 2019, 8:44pm

Hi Dean,

Tim is out today but i was discussing this issue with him yesterday.

We have reached out to our counterparts in Belgium who pointed out you have posted to the tech forum too. They will be looking into it Monday and getting back there.
In regards to server side logs the interest I think would be in seeing what the file look like when sent from the Flexy vs Postman. All we can see from the flexy and postman is that we sent the request and the server is responding with 204. What is the server doing or not doing with the file it did get? What is the file that it received? Is it getting the same file? If not what is different? Do you get any errors on the server when it processes the file from the Flexy that you do not get with the post man?

Deryck

Dean_Nederveld · November 1, 2019, 9:15pm

@deryck_hms, thanks for the update. Calling in the cavalry is much appreciated. I did post on the eWON forum and had a brief interaction with Simon but this forum is where I have submitted the most detailed posts.

The server being posted to is a Linux instance on AWS running InfluxDB. So far everything I have done works well except for the writing of the text file to the database.

I am new to InfluxDB so getting some of the information is a bit time consuming because of the learning curve. One thing I am trying to do now if routing the API request through something before it is sent to the server, in the hope that I can see more information by that means. I tried using the proxy settings on the eWON to route the REQUESTHTTPX command to my laptop but haven’t had success. I am also trying to find another dummy server that can accept API requests to see how it handles things.