As I read the datasheet for the DHT11 a few of things jump out.
1. The resolution for the Humidity is 1%
2. The resolution for the Temperature is 1 degC.

The datastream is 40 bits long.
8bit integral RH data + 8bit decimal RH data + 8bit integral T data + 8bit decimal T data + 8bit check sum.
The algorithm you are implementing stores the bitstream into the dht array like this.
dht[4] = integral RH data
dht[3] = decimal RH data
dht[2] = integral T data
dht[1] = decimal T data
dht[0] = checksum

So the Hex data values you showed earlier "38 00 16 00 22" should be this
dht[0] = checksum = $38 (56 decimal) $00 + $16 + $00 + $22
dht[1] = decimal T data = $00 (this is the decimal portion of the Temperature, which will always be 0)
dht[2] = integral T data = $16 (22 decimal)
dht[3] = decimal RH data = 00 (this is the decimal portion of the Humidty, which will always be 0)
dht[4] = integral RH data = $22 (34 decimal)

I believe you do not need to perform any calculations to convert the RH data to a percentage.
That is what is giving you the strange results.
The calculation "#hum/10" is taking the hum variable and dividing it by 10.
The issue as I see it is that using the hum variable is not necessary and giving you the incorrect values.
PBP treats word variables as Little Endian numbers meaning the low order byte is stored first then the high order byte.
So when hum = _dht+3 and hum is declared as a word, PBP is looking at dht[3] & dht[4] as the word variable hum.
This would be $00 $22, but since we are looking at Little Endian numbers PBP will use dht[4] as the high byte and dht[3] as the low byte.
This makes hum = $2200 (8704 dec), which will result in 870 decimal when you divide 8704 by 10 and a result of 4 with 8704 // 10.
Making your output equal 870.4 as you say.

For the RH percentage you simply need to display the value in dht[4], something like this.
lcdout $FE, $94, dec dht[4], "%"
You do not need the ".",dec1 stuff since the DHT11 does not report a decimal place for RH or Temperature because of the resolution of the sensor.

As for the Temperature, if you are going to just use it as a Celsius number, there is no need for any calculation or conversion either.
Just simply display the value in dht[2]. E.g. lcdout $FE, $94, dec dht[2].

Hope this helps.