collect data (temperature & humidity) from DHT22 into csv and sending them over the net

[malaga]

hello dear fellows, 

 

first of all: i am so glad to be here at this great place - this forum is so a great place - and i have gotten so much help here - and you all encouraged me ever and ever to 

 

a. think over my plans

b. re-design my plans 

c. re-work the concepts. 

d. and to go ahead - 

 

 

thanks for encouraging me.  - this is a awesome place. 

 

hope youre all right and everything goes well at your hometown. Hopefully youre well. 

 

Currently i work on a little SBC (Banana-Pi) project. I want to learn Python, and want to do this with some SBC (with Banana Pi and  and also arduino-projects. 

 

Today: i want  to get data (temperature and humidity) from DHT22 into CSV and sending them over the net

I am trying to have the BananaPi doing some measurements 

 

a. read temperature and humidity from a Adafruit DHT22 sensor, 
b. take all the data and writing them into a .csv file, 

 

Allready i have a setup to read the sensor and print it to the csv-file. 

 

note: the formate of the CSV-file data: it should look like so:

Temperature,Humidity, Date,Time
",6,5,.,6,*,F,",",7,1,.,1,%,"0,6,/,3,4,/,4,0,",",0,6,:,1,4,.,7,2,"

 

some background and preliminary-thoughts:  

a. why i choose the DHT22: well, basically the DHT (DHT stands for Digital Humidity & Temperature) Those sensors are pretty cheap 
- and we can do alot with them: they are really low cost digital sensors with capacitive humidity sensors and thermistors are just great for all measures of the surrounding air. Very good for me to go the first steps in using them with Python.  A great advantage is that the sensors handle analog to digital conversion very nicely and subsequently also provide a 1-wire interface.  So here we go: 

 

import os
import time
import csv
import Adafruit_DHT

DHT_SENSOR = Adafruit_DHT.DHT22
DHT_PIN = 4

def read_sensor():
#Read data from sensor
    h, t = Adafruit_DHT.read_retry(DHT_SENSOR, DHT_PIN)
    
    #Convert from C to F
    t = t * 9/5.0 + 32
    return (t,h)

    if humidity is not None and temperature is not None:
        print("Temp={0:0.1f}*C  Humidity={1:0.1f}%".format(temperature, humidity))

# note; i use this little infinite loop by using the “while TRUE:” code. i hope that i can do this here. 
# what is aimed: this little loop will continually run until a user tries to kills the script.
# right at the beginning of every single loop, we utilize the Adafruit DHT lib to retrieve all the data: a. humidity and b. temperature

    else:
        print("Failed to retrieve data from humidity sensor")

f=open("the_data_logger.csv", "a+")

#Write to CSV file

#If the file is empty write headers
if os.stat('/home/banana/the_data_logger.csv').st_size == 0:
        f.write('Temperature,Humidity\r\n',Date,Time)

wc=csv.writer(f)

#Take 10 measurments and write them to the csv file
for x in range(10):
    h, t = read_sensor()
    wc.writerow('{0},{1},{2:0.1f}*F,{3:0.1f}%\r\n'.format(t, h,time.strftime('%m/%d/%y'),time.strftime('%H:%M.%S'))
    time.sleep(3)
  
    
f.close()

 

Well the input into the csv is a little weird but I think it should work just.

What i want to do next is to take the calc-data and send it automatically over the network to my PC; After this i should be able to input that data into a google sheet for data interpretation. 

 

But this is at the moment a bit tricky.  I will muse about this job - the sending of the data over the network. 

 

My guess: guess that i can do this with ssmpt (or msmpt) while using a gmail-account; amd besides this i should use mailutils  

so far my current datat and thoughts. Now i am going to test the code for storing the data in the csv-file. 
And then i am going to make up my mind how to send the data over the net - to my pc. 

have a great day. 

your malaga **smile**
 

[arox]

I don't understand your point.

 

The problem when designing a distributed system is to ensure reliability when systems crash, reboot or the network disconnects. For systems like SBC or microcontrollers, you have to detect crash and assure recovery, maintenance and debugging on system without screen and keyboard - and avoid corrupted storage problems. For network, you must provide reconnect procedures (witch also suppose that the disconnection has been detected), frequents with over-the-air networks, or in case of peer restart.

 

I use (and you were aiming to do so ?) MQTT with a local broker for that reason. I first so that others used it on this forum and initially thought I didn't need it. But I changed my mind : MQTT allows to deploy a multiple "publisher" / "subscriber" loosely connected network with a trivial message protocol and some option to control reliability of message delivery. Even with that, you will have to carefully design your agents to handle crash, disconnections, sensor errors or network failures. At least, I have found that the Eclipse MQTT broker "mosquitto" is reliable.

 

Basically you can handle MQTT disconnects at application level with a 15 lines of python code MQTT client (if you don't bother to have some samples lost). Sensors error handling is more difficult (often needs reboot). Wifi problems depends on drivers and hardware (so carefully choose the SBC and wifi controller). To ensure reliability of unattended systems connected by wifi is even more difficult in particular with SD cards, rw root fs and OS targeted at graphical UI use case ...

 

I am not sure smtp is a good idea for IoT. It is reliable but slow. MQTT is a messaging protocol with open subscription (no recipient list) and 1 second latency if you implement your broker.  Much more like a "software bus". Ideal in a local network behind a firewall were you don't even need to bother with credentials. If you like complex and heavy solutions or like to be worked up, try Nodered in a Docker container.

 

Well, that is just my point of view ... but I have lost track of the number of agents at my home for a long time. Fortunately, bugs (hardware, software or design) remind me of some regularly.

[lanefu]

to @arox's point.. MQTT is a good fit for this.  

Mosquito is easy to install.

Here's an example of doing something similar.. with a different library

 

Spoiler

import OPi.GPIO as gpio
import logging

#import RPi.GPIO as GPIO
import dht22
import time
import datetime
import paho.mqtt.client as mqtt
import os
#import config
import time
import sys, getopt
import logging
import queue
import random
import sys
from configparser import ConfigParser
import json

def on_connect(client, userdata, flags, rc):
   if rc==0:
      print("connected ok")

config = ConfigParser(delimiters=('=', ))
config.read('config.ini')

topic = config['mqtt'].get('topic', 'temperature/dht22')
decim_digits = config['sensor'].getint('decimal_digits', 2)
sleep_time = config['sensor'].getint('interval', 60)

decim_digits = config['sensor'].getint('decimal_digits', 2)
sleep_time = config['sensor'].getint('interval', 60)

client = mqtt.Client()
client.username_pw_set(config['mqtt'].get('user'), password=config['mqtt'].get('password'))
client.on_connect = on_connect
client.connect(config['mqtt'].get('hostname', 'homeassistant'),
               config['mqtt'].getint('port', 1883),
               config['mqtt'].getint('timeout', 60))
client.loop_start()

# initialize GPIO
#gpio.setwarnings(False)
gpio.setboard(gpio.PRIME)
gpio.setmode(gpio.SOC)
#PIN2 = port.PA13
pin = gpio.PA + 10
#gpio.cleanup()

gpio.setup(pin, gpio.IN, pull_up_down=gpio.PUD_OFF)
# read data using pin 14
instance = dht22.DHT22(pin)

while True:
    result = instance.read()
    if result.is_valid():
        fResult = ( result.temperature * (9 / 5) + 32)
        print("Last valid input: " + str(datetime.datetime.now()))
        print("Temperature: %.2f C" % result.temperature)
       #im an American
        print("Temperature: %.2f F" % fResult)
        print("Humidity: %.2f %%\n" % result.humidity)
        client.publish(topic + "/humidity", str(result.humidity))
        client.publish(topic + "/temperature", f"{fResult:.2f}")
    time.sleep(15)
    

 

 

[malaga]

hello dear Arox, hello dear lanefu, 

first of all: many many thanks for the answer and for the headsup. 

Arox:

Quote

The problem when designing a distributed system is to ensure reliability when systems crash, reboot or the network disconnects. For systems like SBC or microcontrollers, you have to detect crash and assure recovery, maintenance and debugging on system without screen and keyboard - and avoid corrupted storage problems. For network, you must provide reconnect procedures (witch also suppose that the disconnection has been detected), frequents with over-the-air networks, or in case of peer restart.

thank you Arox - youre right. 

 

Lanefu:

Quote

MQTT is a good fit for this.  
Mosquito is easy to install.

i am not so familiar with MQTT - protocolls. But i guess youre  right. We can do this with MQTT too. 

we could read the DHT-sensor on BananaPi and mqtt the results.

But what if we do it - in other words we can read a sensor like a DHT22 or similar and mqtt the results - 
(sending the data to GPIO pins) 

 

So - we can read a the sensor (DHT22) and then we can MQTT the result to a MQTT broker (the broker could be 

 

a. Banana Pi/ OrangePi / RaspBerryPi or similar - (or even better):
b. ESP 32 or 8266

 

that said: Arox and Lanefu - youre right: the MQTT broker could be the banana pi (or a raspberrypi or similar) on which i connect the DHT22 to.
so if I wanted I could read it directly, but just in case I wanted to read it from another raspberry. To sume up: MQTT is a pretty nice solution.

i try to work it out. 

 

i come back and if i have a bit of a structured plan. 

 

Again - many many thanks for the great ideas - and for encouraging me to do this in MQTT.

 

this is a awesome forum. 

 

greetings 

 

Malaga

 

update: well it could look like this...

 

 

 

                                         +----------------------------------+                       
                                         |                                  |                       
                  						 |                                  |                       
             						     |    MQTT- Broker                  |                       
 						                 |                                  |                       
                                         |                                  |                       
                                         |                                  |                       
                                         |                                  |                       
                                         |           Topic                  |                       
                                         |                                  |                       
                                         |                                  |                       
+--------------------------+             |   +--------------------------+   |   publish             
|  Temperature             |  subscribe  |   |         temperature      |---------------------------
|                          |-----------------|                          |   |                       
|                          |             |   +--------------------------+   |                       
+--------------------------+             |                                  |                       
                     +                   |                                  |                       
                                         |                                  |                       
                                         |                                  |                       
+--------------------------+ subscribe   |   +--------------------------+   |   publish             
|   Humidity               |------------------       humidity           |---------------------------
|                          |             |   |                          |   |                       
|                          |             |   +--------------------------+   |                       
+--------------------------+             |                                  |                       
                                         |                                  |                       
                                         |                                  |                       
+-------------------------+              |   +--------------------------+   |   publish             
|  more data              | subscribe    |   |      more data           |------------------------   
|                         |------------------|                          |   |                       
|                         |              |   +--------------------------+   |                       
|                         |              |                                  |                       
+-------------------------+              |                                  |                       
                                         +----------------------------------+                       

 

[arox]

"i am not so familiar with MQTT - protocolls."

 

MQTT is the most simplistic protocol that can exist : you say you want to send (publish) or receive (subscribe), about what (topic) and what (message). The only problem you can encounter in python is to provide an event loop for receiving messages. Messages are handled by a function defined as a handler. You just need to call "client.loop_forever()" or "client.loop_start()" at interval in another loop if you want to do other stuff in a subscribing client.

 

The protocols that handle sensors are much more problematic : they need precise timing to send receive/receive data on gpio, which is not so easy in "not real time" OS and then with SBC. So microcontrollers are great for sensor driver and MQTT publisher.

 

SBC are great for broker because you will want a log to see connection / disconnection events and you need storage if you want persistent messages (*). But the most basic SBC do the trick : my brocker run on a tiny H3 nanopi which handles all communication functions (firewall, http proxy and filter, mail relay, DNS, NTP server ...) Mosquitto is perhaps the more easy to install and reliable server I have ever used.

 

The fun resides in MQTT subscriber : you can analyse data, command electrical circuits with 2N7000 Mosfet and arduino relay boards, drive LED strips with PWM, stepper motors or anything you want for a couple of bucks, a soldering iron and a lot of time ... Of course, SBC are more suitable for data analysis and microcontrollers for hardware interfacing.

 

All in all, I spend 5 more time for hardware than for software and 5 more time for software than for design. And no time for "protocols" because they are implemented in drivers or libraries. So don't dwell on it but go for it !

 

(*) there are just 2 options in MQTT 3.11 : persistent messages keep messages and deliver them to new clients that connect and subscribe after publication, qos indicates whether you can afford to drop a message in case of delivery failure, accept duplication or want a unique delivery. And one more option to adjust the "keep alive" / disconnect time (an "integrated ping" between client and broker that I never bothered to change).