On the server side of this can you tell us what response is given for an individual string of data vs when you send multiple strings of data?
Below is the setup on the Postman Params tab:
Below is the setup on the Body tab:
Where datatopost is a text file on my laptop identical to the text file on my device that I am trying to send to my database.
Below are the header responses:
The response 204 No Content plus checking the database to see that the data was successfully written is my indication that the Postman command executed properly.
You asked āOn the server side of this can you tell us what response is given for an individual string of data vs when you send multiple strings of data?ā I will have to do some research on Influx to see if anything is noted or logged on the server side.
@Tim_hms, I believe I was able to get some information from the server side logs for a particular REQUESTHTTPX command I executed. I modified my script so I could execute just one REQUESTHTTPX command to write a text file of data to the database and then stop. Here is the response from the database server as shown in the BASIC IDE console of my eWON:
This is the server side log entry for the above command:
[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:13:53:10 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "-" f1040bde-fcae-11e9-9bbe-0e5b008f42bc 81
The idās and the timestamps are the same so thatās the information from the transaction. I do note that the file name is not in the log entry on the server side.
Below is the response from the database server as shown in the BASIC IDE console of my eWON of sending a single string of data (in the S4 Post-Data switch of the REQUESTHTTPX command):
This is the server side log entry for the above command:
[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:14:05:42 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "-" b141aae6-fcb0-11e9-b009-0e5b008f42bc 12457
Finally, here is the same information from write a text file of data to the database from Postman:
[httpd] <<IP ADDRESS HIDDEN>> - - [01/Nov/2019:14:11:20 +0000] "POST /write?db=import HTTP/1.1" 204 0 "-" "PostmanRuntime/7.19.0" 7a856350-fcb1-11e9-ba77-0e5b008f42bc 334349
Hi Dean,
Tim is out today but i was discussing this issue with him yesterday.
-
We have reached out to our counterparts in Belgium who pointed out you have posted to the tech forum too. They will be looking into it Monday and getting back there.
-
In regards to server side logs the interest I think would be in seeing what the file look like when sent from the Flexy vs Postman. All we can see from the flexy and postman is that we sent the request and the server is responding with 204. What is the server doing or not doing with the file it did get? What is the file that it received? Is it getting the same file? If not what is different? Do you get any errors on the server when it processes the file from the Flexy that you do not get with the post man?
Deryck
@deryck_hms, thanks for the update. Calling in the cavalry is much appreciated. I did post on the eWON forum and had a brief interaction with Simon but this forum is where I have submitted the most detailed posts.
The server being posted to is a Linux instance on AWS running InfluxDB. So far everything I have done works well except for the writing of the text file to the database.
I am new to InfluxDB so getting some of the information is a bit time consuming because of the learning curve. One thing I am trying to do now if routing the API request through something before it is sent to the server, in the hope that I can see more information by that means. I tried using the proxy settings on the eWON to route the REQUESTHTTPX command to my laptop but havenāt had success. I am also trying to find another dummy server that can accept API requests to see how it handles things.
I think looking into the server side is going to be required. You are saying you are sending the same file with postman and with the Flexy. the server is responding with the same response 204, saying it was successful. From that point it should be getting processes on the server/ influxDB. Where it is processing 1 and not the other it would be helpful to know why. From there we can trace it back into the Flexy to see what it is doing to cause that problem.
My guess is that the file sent though postman is sent as one file type and the flexy is sending another.
We can try to tackle it directly from the Flexy but we donāt know what is wrong. Knowing exactly what is wrong should make troubleshooting a whole lot easier. Does the influx DB have a log file?
Can you share a copy of the .bin file you are sending?
Perhaps generate it right on the Flexy and send that and the post man one.
I am working this from both sides, the eWON and InfluxDB. Do you know of another location to which I can send the REQUESTHTTPX POST request with the file to see if the file is being sent from the eWON? Or a way I can route it to a proxy first so I can check the status at the proxy to see what makes it to that point?
At the moment the InfluxDB log just shows the request was made and the response that it appears to have been correctly processed. I get not discernible information between when the file is sent through Postman or the Flexy. I tried apitester.com and that works too.
I also used InfluxDBās Chronograf software, running on the same AWS Linux instance as InfluxDB, which allowed me to upload the text file from the Flexy SD card, with success.
I also just did the same through Postman, by telling Postman to get the file from the Flexy SD card. What I donāt know is whether Postman is transmitting it directly from the SD card or creating a temporary file on my laptop and then sending that file.
This is the file on the Flexy SD card to which my BASIC IDE script is writing the data: datatopost (10.7 KB)
Hi Dean,
Hereās what I heard back from my colleague in Belgium:
or the way the Flexy posts data is not taken in account by the server for a reason
Maybe he should use:
Content-Type: application/x-www-form-urlencoded as header value because in the Curl call given as example in the manual, they use --data-binary
Regarding āDo you know of another location to which I can send the REQUESTHTTPX POST request with the file to see if the file is being sent from the eWON?ā
they can use the ewonsupport.biz/post
@Tim_hms, thanks for the suggestions. Here are the results:
I changed the content type header as recommended. The command still did not write any data to the database.
I left the content type as recommended and changed the url$ variable to http://ewonsupport.biz/post/ and executed the command. I received a 200 status code in response which I presume means the command sent the file (datatpost.txt) to the desired location, although I do not have a way of verifying.
Below is the request header as formulated by Postman and the response header from the database server:
POST http://<<IP_ADDRESS_DB-SERVER>>:8086/write?db=import
Network
addresses: {ā¦}
local: {ā¦}
address: "<<IP_ADDRESS_DEVICE>>"
family: "IPv4"
port: 52282
remote: {ā¦}
address: "<<IP_ADDRESS_DB-SERVER>>"
family: "IPv4"
port: 8086
Request Headers
User-Agent: "PostmanRuntime/7.19.0"
Accept: "*/*"
Cache-Control: "no-cache"
Postman-Token: "55751d0b-3f3e-4212-afe8-ccb529ac1d19"
Host: "<<IP_ADDRESS_DB-SERVER>>:8086"
Accept-Encoding: "gzip, deflate"
Content-Length: 2425
Content-Type:
Connection: "keep-alive"
Request Body
content: {ā¦}
src: "/C:/Users/<<USER_NAME>>/AppData/Local/Microsoft/Windows/Temporary Internet Files/Content.IE5/MB5RP42O/datatopost[1]"
Response Headers
Content-Type: "application/json"
Request-Id: "cb1a455c-ffd4-11e9-a9fc-0e5b008f42bc"
X-Influxdb-Build: "OSS"
X-Influxdb-Version: "1.7.9"
X-Request-Id: "cb1a455c-ffd4-11e9-a9fc-0e5b008f42bc"
Date: "Tue, 05 Nov 2019 14:01:41 GMT"
A few things I note:
-
The Content-Type in the request header from Postman is blank. There is an item buried in the Request Body section under Request Body > content: __readablestate: > defaultEncoding: āutf8ā_, which is, perhaps, how Postman identifies the payload as a binary file.
-
Under Request Body > content: > src:, prior to sending the file Postman seems to upload the file from the eWON to a temporary file on my laptop. I donāt know if anything about the nature of the file changes but I doubt it. In fact, I would expect the file posted directly from the Linux-formatted SD card on the eWON to the Linux-based database server to be a ācleanerā transmission of the file.
I also successfully tested another REST API location with Postman. However, when I tested from the eWON nothing was transmitted. I am beginning to believe the REQUESTHTTPX command is not actually sending the file.
I have now tested this on an eWON running firmwre 14.0PR and another running 13.3s0.
@Tim_hms, another updateā¦I highly suspect the REQUESTHTTPX command is ignoring the EBD text I have placed in the S5 field.
I have found another site to test what is being sent by the REQUESTHTTPX command, requestbin.com. The site allows a temporary REST API to be configured that can be used in the BASIC script. You could easily set one up and test the script below by modifying specific parameters to match your eWON and the temporary site request.bin configures for you. When you configure the temporary site make sure to uncheck Private to make things a bit easier to test.
This is the script I am testing:
url$ = "https://enu0qp1lxyg.x.pipedream.net"
DataToPost$ = "DEVELOPMENT_DATA,unit=DEVELOPMENT val1=582.00 1572275711000000000"
method$ = "POST"
header$ = "Content-Type=application/x-www-form-urlencoded"
REQUESTHTTPX url$, method$, header$, "", "[$dtUF $ftB $uf/usr/sdext/datatopost $fndatatopost.txt]"
The above code seems to do nothing. I would expect the datatopost file on the eWON to be uploaded as datatopost.txt but it is not. This is what I see on the requestbin.com site, nothing more:
If I make the S4 parameter the variable DataToPost$, then I see this on requestbin.com:
So, anything I place in the S4 Post-Data parameter is transmitted, but the S5 File-Data parameter always seems to be ignored.
Again, you can easily set up a test on your eWON using the information in this message, and see if you can transmit a file from your eWON.
Hi Dean,
Iām still working on this one with my colleague in Belgium but Iāll send him the updates you sent me and see if he has some new ideas
Hi Dean,
I was talking with my colleague in Belgium and he was saying that FileData is something special that needs to be supported on the server side. He said he did see that you successfully tested it with https://ewonsupport.biz/post/ though
@Tim_hms, I believe the file I uploaded was through using the https://ewonsupport.biz/post/ page directly, not by accessing that page using the REQUESTHTTPX command in the script.
I just performed the same routine by first uploading the file directly from the page and calling it datatopost_via_page.txt. I see that it did upload via that method.
I then tried it from the script using the REQUESTHTTPX command the the url https://ewonsupport.biz/post/. The is the response I see in the console of the BASIC script:
I assume the file transferred but I have no way of verifying.
Hi Dean,
We may have just found something, the File transfer (S5) does not work if you change the header of the request (S3)
he tried this:
REQUESTHTTPX āhttps://envpodn891j4g.x.pipedream.netā, āPOSTā, āā, āā, āfile=[$dtEV]ā, āā
then got this response:
It also looks like you cannot push raw data (S4) and a file (S5) at the same time, so maybe try just pushing a file and see if that goes through?
@Tim_hms, great! I just tried this command from the script:
REQUESTHTTPX url$, method$, "", "", "file=[$dtUF $ftB $uf/usr/sdext/datatopost_via_script.txt $fndatatopost_via_script.txt]", "", ""
With url$=āhttps://enze2d1q7ai2a.x.pipedream.netā and method$=āPOSTā. Here is the result on that test server:
Thatās the first time Iāve seen the file contents transfer from the script.
For grins I tested it on another site and received a similar result.
And thenā¦I tested the script to the database server, and it worked! The data ingested as expected. I tested it first with a small file of 25 points that had a txt file extension, then I tested it with a larger file of over 23,000 lines of data (about 30 hours of collected data) with no file extension, and it worked too!!
REQUESTHTTPX url$, method$, "", "", "file=[$dtUF $ftB $uf/usr/sdext/datatopost_old $fndatatopost_old.bin]", "", ""
The one thing I note about the format of the command that finally worked is the absence of a header, the S3 parameter is null (""). I thought I tried this in the many attempts to get it to work but I suppose I just didnāt have the right format of other aspects of the command.
One thing that is odd is that the response from the database server gives the impression the data didnāt ingest. The response includes text such as āmissing fields unable to parseā, āinvalid field format unable to parseā, but the data fully ingests, which is the ultimate goal. Perhaps the file doesnāt have an EOF character or something the database server is looking for to know it reached the end of the data.
Nevertheless, itās working and I really appreciate sticking with me on this one. Hopefully what is shared in this thread will be helpful to others.
Thanks much!
Thanks for the work on this one Dean!
It helped us uncover a bug that doesnāt seem to let you post raw data and files at the same time. Glad we got it working for you!
@Tim_hms, one final postscriptā¦
My end game on this has been to post data to the database server when the connection is active and the server responds that the data has been posted. When the connection is interrupted I am saving the string of data to a file on the SD card of the eWON. The file gets appended with new strings of data until the connection is re-established. The script then sends the file of collected data to the server, so that no data goes missing. Finally, the file gets renamed with an _prev suffix so I can see what data should have been transmitted.
Hi Dean is the last question youāre having on how to rename the file with the _prev suffix?