Influx with RequestHttpx

Ewon Support Ewon Flexy
#1

I have successfully written an Influx line protocol string of data to an Influx database using the REQUESTHTTPX command with the POST method.

One thing I am currently testing is writing to Influx a text file containing multiple line protocol strings of data. I have successfully been able to write data to a text file on SD card in the eWON, and then used the Postman app to manually send the text file to Influx which it easily reads into the database.

I am trying to do that same task of writing the file automatically using the REQUESTHTTPX command with the S5 File-Data option. So far the command executes and receive the expected response that the transaction occurred with Influx but the data does not get written to the database.

Still trying…

Data posts to InfluxDB
Data transfer from Flexy to InfluxDB
#2

Hi Dean,

Are you seeing any errors on the eWON’s event logs?

#4

No error in the event logs although some of the events say they are muted. I get the expected response code (204) following the REQUESTHTTPX command to export the file so I don’t know what is keeping the data from being written to the database.

#5

No errors, and I notice no matter what I use for Content-Type header, even gibberish, I still get a response code as if the command executed as expected and wrote data to the database from the file.

#6

Could you send us a backup.tar of the device from ebuddy with the support files included?

#7

MOVED TO STAFF NOTE (610.5 KB)
Please the request file included with this reply. Thank you for your assistance.

#9

Hi Dean,

One thing I noticed on this was that it looks like your device doesn’t seem to have a very strong signal strength. The minimum signal strength we recommend for a strong connection is 18. Do we have a way to try and move the device/antenna to see if we can improve the signal and see if it’s still running into issues?

image

#10

This is device I use for development. It is configured to use Ethernet as the primary interface with cellular as the fallback.

image

#11

Just to check was there a time when the data was successfully being written to the database initially or has there been no data written to it yet? If so could we try and slow down the timer and see if it is still running into issues?

#12

I have been able to repeatedly write a single string of data to the REST API of the database using the REQUESTHTTPX command with the POST method. The database uses the same method to ingest a text file containing several strings of data in the same format as the single string. I have not been able to ingest this file of data via the eWON.

I can copy the file from the eWON to my laptop and, using an app called Postman, I can write this file to the database. In Postman, to write a string of data to the REST API I choose the ‘raw’ option. To write the file of data I choose the ‘binary’ option. I don’t know what Postman does differently when it executes the command but I do know I have to manually enter in the body the single data string when choosing the ‘raw’ option and I am allowed to select the text file of the multiple strings when choosing the ‘binary’ option.

#13

We do have a way to save the data as a binary files in your SD card if that would make it possible to post to Influx.

https://ewonsupport.biz/ebd/

image
image872×351 18.1 KB

#14

@Tim_hms, my script successfully creates and writes to a text file on the SD card, although it’s not using the EBD method. This is the section that writes a single string of data in the format required by the database:

   OPEN "file:" + $DataPostLoc$ FOR BINARY APPEND AS 1
   PUT 1, $DataToPost$ + CHR$(10)
   CLOSE 1

The file is successfully written to, and I can copy that file to my laptop and send it to the database via Postman.

What I cannot do is use the REQUESTHTTPX command to send the file to the database, although I believe I should be able to.

#15

@Tim_hms, I was able to successfully use the SENDMAIL command to email myself the file from the SD card as an attachment,

SENDMAIL “email@gmail.com”, “”, “TEST”, “&[$dtUF$ftT$uf” + $fileToSend$ + “]”

I note, however, that the attached file name is “$dtUF$ftT$uf/usr/sdext/datatopost.txt.txt”. I believe this may be the problem if the database sees all the extra characters other than the simple file name.

#16

I believe you should be able to fix this by using the filename part of the EBD

so you could do something like this $fnyourfilename

#17

@Tim_hms, I agree with adding the $fn switch and was on that track this morning. I first tested it with the EBD helper and the results were as I expected. The filename used to save to my laptop with the $fn switch was the simple filename I expected.

I also tried using the SENDMAIL command but I had an issue with AWS Simple Email Service (SES) requiring a STARTTLS instruction, so I switched to relay.talk2m.com for testing and did prove receipt of an email with the data file attached with the simple file name ‘datatopost.txt’ without the extraneous EBD characters. Progress!

This worked –
SENDMAIL “email@gmail.com”, “”, “TEST”, “&[$dtUF$ftB$uf/usr/sdext/datatopost$fndatatopost.txt]”

However, the REQUESTHTTPX command, now with the $fn switch, still isn’t sending the POST to the database in a manner in which the database can ingest the data.

This still isn’t fully working –
REQUESTHTTPX url$, method$, header$, “”, “[$dtUF$ftB$uf/usr/sdext/datatopost$fndatatopost.txt]”

#18

I dumbed down the script so I could just force a single REQUESTHTTPX command each time I stop and start the script. It seems no matter what I place in the header, so long as it is not null and it starts with Content-Type=, the command executes and I receive the anticipated response from the database server. I can even leave the header as Content-Type= with nothing after the equal sign and it seems to execute but no data is written.

POST_DATA:
FileSize% = FS "size", "/usr/sdext/datatopost"
url$ = "http://IP_ADDRESS:PORT/write?db=import"
method$ = "POST"
//header$ = "Content-Type=text/plain;charset=utf-8"
header$ = "Content-Type=X"
REQUESTHTTPX url$, method$, header$, "", "&[$dtUF$ftB$uf/usr/sdext/datatopost$fndatatopost.bin]"
actionID% = GETSYS PRG,"ACTIONID"
PRINT "Request actionID: "; actionID%
ONSTATUS "GOTO ON_EVENT" 
END

ON_EVENT:
eventID% = GETSYS PRG,"EVTINFO"
IF (eventID% = actionID%) THEN
  SETSYS PRG,"ACTIONID",eventID%
  stat% = GETSYS PRG,"ACTIONSTAT"
  IF (stat% = 0) THEN
    GOTO RESPONSE
  ELSE
    PRINT "Error: " + STR$ stat%
  ENDIF
ENDIF
END

RESPONSE:
a$ = RESPONSEHTTPX "STATUSCODE"
$LogText$ = "***POST FILE ** STATUSCODE: " + a$ + " ACTIONSTAT: " + STR$ stat%
b$ = RESPONSEHTTPX "HEADER"
PRINT "***all headers: " + b$;
$FileSize% = FS "size", "/usr/sdext/datatopost"
$LogText$ = $LogText$ + " FILESIZE: " + STR$ $FileSize%
PRINT $LogText$
LOGEVENT $LogText$
END

Each time I execute the above, I receive this response from the database server:

image

When I send the same file via the Postman app I receive this response:

image
image711×293 6.54 KB

The one thing I notice as different between the responses is the response to the eWON includes connection: close whereas that is not include in the response to Postman.

#19

I changed the script a bit so the posting of the file of data would wait five (5) seconds after the posting of the string of data. I did this so I can compare the response from the database server from each REQUESTHTTPX command.

This is the response from the execution of the REQUESTHTTPX command to send a single string of data to the database server, and a successful writing or ingesting of the data into the database:

image

I have been able to execute this command successfully over and again.

This is the response from the execution of the REQUESTHTTPX command to send a text file of multiple strings of data to the database server, but the data is not successfully written or ingested into the database:

image

Note, the responses are virtually identical even though only the first one is successful.

#20

Hi Dean,

I think I might need to escalate this to one of my colleagues in Belgium to get a better answer on why we might be running into this issue

#21

@Tim_hms, Thank you. I look forward to the comments from you and your colleague(s).

#22

Can you send us what you’re doing in Postman?