Influx with RequestHttpx


#23

On the server side of this can you tell us what response is given for an individual string of data vs when you send multiple strings of data?


#24

Below is the setup on the Postman Params tab:
image

Below is the setup on the Body tab:
image

Where datatopost is a text file on my laptop identical to the text file on my device that I am trying to send to my database.

Below are the header responses:

The response 204 No Content plus checking the database to see that the data was successfully written is my indication that the Postman command executed properly.

You asked “On the server side of this can you tell us what response is given for an individual string of data vs when you send multiple strings of data?” I will have to do some research on Influx to see if anything is noted or logged on the server side.


#25

@Tim_hms, I believe I was able to get some information from the server side logs for a particular REQUESTHTTPX command I executed. I modified my script so I could execute just one REQUESTHTTPX command to write a text file of data to the database and then stop. Here is the response from the database server as shown in the BASIC IDE console of my eWON:

image

This is the server side log entry for the above command:

[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:13:53:10 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "-" f1040bde-fcae-11e9-9bbe-0e5b008f42bc 81

The id’s and the timestamps are the same so that’s the information from the transaction. I do note that the file name is not in the log entry on the server side.

Below is the response from the database server as shown in the BASIC IDE console of my eWON of sending a single string of data (in the S4 Post-Data switch of the REQUESTHTTPX command):

image

This is the server side log entry for the above command:

[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:14:05:42 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "-" b141aae6-fcb0-11e9-b009-0e5b008f42bc 12457

Finally, here is the same information from write a text file of data to the database from Postman:

[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:14:11:20 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "PostmanRuntime/7.19.0" 7a856350-fcb1-11e9-ba77-0e5b008f42bc 334349

#26

Hi Dean,

Tim is out today but i was discussing this issue with him yesterday.

  1. We have reached out to our counterparts in Belgium who pointed out you have posted to the tech forum too. They will be looking into it Monday and getting back there.

  2. In regards to server side logs the interest I think would be in seeing what the file look like when sent from the Flexy vs Postman. All we can see from the flexy and postman is that we sent the request and the server is responding with 204. What is the server doing or not doing with the file it did get? What is the file that it received? Is it getting the same file? If not what is different? Do you get any errors on the server when it processes the file from the Flexy that you do not get with the post man?

Deryck


#27

@deryck_hms, thanks for the update. Calling in the cavalry is much appreciated. I did post on the eWON forum and had a brief interaction with Simon but this forum is where I have submitted the most detailed posts.

The server being posted to is a Linux instance on AWS running InfluxDB. So far everything I have done works well except for the writing of the text file to the database.

I am new to InfluxDB so getting some of the information is a bit time consuming because of the learning curve. One thing I am trying to do now if routing the API request through something before it is sent to the server, in the hope that I can see more information by that means. I tried using the proxy settings on the eWON to route the REQUESTHTTPX command to my laptop but haven’t had success. I am also trying to find another dummy server that can accept API requests to see how it handles things.


#28

I think looking into the server side is going to be required. You are saying you are sending the same file with postman and with the Flexy. the server is responding with the same response 204, saying it was successful. From that point it should be getting processes on the server/ influxDB. Where it is processing 1 and not the other it would be helpful to know why. From there we can trace it back into the Flexy to see what it is doing to cause that problem.

My guess is that the file sent though postman is sent as one file type and the flexy is sending another.

We can try to tackle it directly from the Flexy but we don’t know what is wrong. Knowing exactly what is wrong should make troubleshooting a whole lot easier. Does the influx DB have a log file?


#29

Can you share a copy of the .bin file you are sending?
Perhaps generate it right on the Flexy and send that and the post man one.


#30

I am working this from both sides, the eWON and InfluxDB. Do you know of another location to which I can send the REQUESTHTTPX POST request with the file to see if the file is being sent from the eWON? Or a way I can route it to a proxy first so I can check the status at the proxy to see what makes it to that point?

At the moment the InfluxDB log just shows the request was made and the response that it appears to have been correctly processed. I get not discernible information between when the file is sent through Postman or the Flexy. I tried apitester.com and that works too.

I also used InfluxDB’s Chronograf software, running on the same AWS Linux instance as InfluxDB, which allowed me to upload the text file from the Flexy SD card, with success.

I also just did the same through Postman, by telling Postman to get the file from the Flexy SD card. What I don’t know is whether Postman is transmitting it directly from the SD card or creating a temporary file on my laptop and then sending that file.

This is the file on the Flexy SD card to which my BASIC IDE script is writing the data: datatopost (10.7 KB)


#31

Hi Dean,

Here’s what I heard back from my colleague in Belgium:

or the way the Flexy posts data is not taken in account by the server for a reason

Maybe he should use:

Content-Type: application/x-www-form-urlencoded as header value because in the Curl call given as example in the manual, they use --data-binary

Regarding “Do you know of another location to which I can send the REQUESTHTTPX POST request with the file to see if the file is being sent from the eWON?”

they can use the ewonsupport.biz/post


#32

@Tim_hms, thanks for the suggestions. Here are the results:

I changed the content type header as recommended. The command still did not write any data to the database.

I left the content type as recommended and changed the url$ variable to http://ewonsupport.biz/post/ and executed the command. I received a 200 status code in response which I presume means the command sent the file (datatpost.txt) to the desired location, although I do not have a way of verifying.


#33

Below is the request header as formulated by Postman and the response header from the database server:

POST http://<<IP_ADDRESS_DB-SERVER>>:8086/write?db=import
  Network
    addresses: {…}
      local: {…}
        address: "<<IP_ADDRESS_DEVICE>>"
        family: "IPv4"
        port: 52282
      remote: {…}
        address: "<<IP_ADDRESS_DB-SERVER>>"
        family: "IPv4"
        port: 8086
  Request Headers
    User-Agent: "PostmanRuntime/7.19.0"
    Accept: "*/*"
    Cache-Control: "no-cache"
    Postman-Token: "55751d0b-3f3e-4212-afe8-ccb529ac1d19"
    Host: "<<IP_ADDRESS_DB-SERVER>>:8086"
    Accept-Encoding: "gzip, deflate"
    Content-Length: 2425
    Content-Type:
    Connection: "keep-alive"
  Request Body
    content: {…}
    src: "/C:/Users/<<USER_NAME>>/AppData/Local/Microsoft/Windows/Temporary Internet Files/Content.IE5/MB5RP42O/datatopost[1]"
  Response Headers
    Content-Type: "application/json"
    Request-Id: "cb1a455c-ffd4-11e9-a9fc-0e5b008f42bc"
    X-Influxdb-Build: "OSS"
    X-Influxdb-Version: "1.7.9"
    X-Request-Id: "cb1a455c-ffd4-11e9-a9fc-0e5b008f42bc"
    Date: "Tue, 05 Nov 2019 14:01:41 GMT"

A few things I note:

  1. The Content-Type in the request header from Postman is blank. There is an item buried in the Request Body section under Request Body > content: __readablestate: > defaultEncoding: “utf8”_, which is, perhaps, how Postman identifies the payload as a binary file.

  2. Under Request Body > content: > src:, prior to sending the file Postman seems to upload the file from the eWON to a temporary file on my laptop. I don’t know if anything about the nature of the file changes but I doubt it. In fact, I would expect the file posted directly from the Linux-formatted SD card on the eWON to the Linux-based database server to be a “cleaner” transmission of the file.

I also successfully tested another REST API location with Postman. However, when I tested from the eWON nothing was transmitted. I am beginning to believe the REQUESTHTTPX command is not actually sending the file.

I have now tested this on an eWON running firmwre 14.0PR and another running 13.3s0.


#34

@Tim_hms, another update…I highly suspect the REQUESTHTTPX command is ignoring the EBD text I have placed in the S5 field.

I have found another site to test what is being sent by the REQUESTHTTPX command, requestbin.com. The site allows a temporary REST API to be configured that can be used in the BASIC script. You could easily set one up and test the script below by modifying specific parameters to match your eWON and the temporary site request.bin configures for you. When you configure the temporary site make sure to uncheck Private to make things a bit easier to test.
image

This is the script I am testing:

url$ = "https://enu0qp1lxyg.x.pipedream.net"
DataToPost$ = "DEVELOPMENT_DATA,unit=DEVELOPMENT val1=582.00 1572275711000000000"
method$ = "POST"
header$ = "Content-Type=application/x-www-form-urlencoded"
REQUESTHTTPX url$, method$, header$, "", "[$dtUF $ftB $uf/usr/sdext/datatopost $fndatatopost.txt]"

The above code seems to do nothing. I would expect the datatopost file on the eWON to be uploaded as datatopost.txt but it is not. This is what I see on the requestbin.com site, nothing more:

image

If I make the S4 parameter the variable DataToPost$, then I see this on requestbin.com:

image

So, anything I place in the S4 Post-Data parameter is transmitted, but the S5 File-Data parameter always seems to be ignored.

Again, you can easily set up a test on your eWON using the information in this message, and see if you can transmit a file from your eWON.


#35

Hi Dean,

I’m still working on this one with my colleague in Belgium but I’ll send him the updates you sent me and see if he has some new ideas


#36

Hi Dean,

I was talking with my colleague in Belgium and he was saying that FileData is something special that needs to be supported on the server side. He said he did see that you successfully tested it with https://ewonsupport.biz/post/ though


#37

@Tim_hms, I believe the file I uploaded was through using the https://ewonsupport.biz/post/ page directly, not by accessing that page using the REQUESTHTTPX command in the script.

I just performed the same routine by first uploading the file directly from the page and calling it datatopost_via_page.txt. I see that it did upload via that method.

I then tried it from the script using the REQUESTHTTPX command the the url https://ewonsupport.biz/post/. The is the response I see in the console of the BASIC script:

image

I assume the file transferred but I have no way of verifying.


#38

Hi Dean,

We may have just found something, the File transfer (S5) does not work if you change the header of the request (S3)

he tried this:
REQUESTHTTPX “https://envpodn891j4g.x.pipedream.net”, “POST”, “”, ‘’, “file=[$dtEV]”, “”

then got this response:
image

It also looks like you cannot push raw data (S4) and a file (S5) at the same time, so maybe try just pushing a file and see if that goes through?


#39

@Tim_hms, great! I just tried this command from the script:

REQUESTHTTPX url$, method$, "", "", "file=[$dtUF $ftB $uf/usr/sdext/datatopost_via_script.txt $fndatatopost_via_script.txt]", "", ""

With url$=“https://enze2d1q7ai2a.x.pipedream.net and method$=“POST”. Here is the result on that test server:

That’s the first time I’ve seen the file contents transfer from the script.

For grins I tested it on another site and received a similar result.

And then…I tested the script to the database server, and it worked! The data ingested as expected. I tested it first with a small file of 25 points that had a txt file extension, then I tested it with a larger file of over 23,000 lines of data (about 30 hours of collected data) with no file extension, and it worked too!!

REQUESTHTTPX url$, method$, "", "", "file=[$dtUF $ftB $uf/usr/sdext/datatopost_old $fndatatopost_old.bin]", "", ""

The one thing I note about the format of the command that finally worked is the absence of a header, the S3 parameter is null (""). I thought I tried this in the many attempts to get it to work but I suppose I just didn’t have the right format of other aspects of the command.

One thing that is odd is that the response from the database server gives the impression the data didn’t ingest. The response includes text such as “missing fields unable to parse”, “invalid field format unable to parse”, but the data fully ingests, which is the ultimate goal. Perhaps the file doesn’t have an EOF character or something the database server is looking for to know it reached the end of the data.

Nevertheless, it’s working and I really appreciate sticking with me on this one. Hopefully what is shared in this thread will be helpful to others.

Thanks much!


#40

Thanks for the work on this one Dean!

It helped us uncover a bug that doesn’t seem to let you post raw data and files at the same time. Glad we got it working for you!


#41

@Tim_hms, one final postscript…

My end game on this has been to post data to the database server when the connection is active and the server responds that the data has been posted. When the connection is interrupted I am saving the string of data to a file on the SD card of the eWON. The file gets appended with new strings of data until the connection is re-established. The script then sends the file of collected data to the server, so that no data goes missing. Finally, the file gets renamed with an _prev suffix so I can see what data should have been transmitted.


#42

Hi Dean is the last question you’re having on how to rename the file with the _prev suffix?