Let's do some order. Mod functions are in the realm of ring theory. I assume you can guess what a ring is. It is good place where you have stuff that behaves as if they were numbers. You can add and subtract that stuff and you can multiply it in a way that distributes over addition.
Given a ring \(R\) we can study its
ideals \(I\subseteq R\). Ideals are special subsets of the ring that behaves as if they were linear subspaces of \(R\) (lines, planes and so on) but somewhat more abstractly than these.
Now, an ideal \(I\) in \(R\) kinda traces out a figure in \(R\) and we can use it to "fold" \(R\) in such a way that the folded ring \(R/I\) has all the points in the figure \(I\subseteq R\) collapsed to zero, so they disappears. More technically, an ideal defines an equivalence relation over the ring.
This folding is called quotienting out \(R\) by the ideal \(I\). Given an ideal \(I\subseteq R\) there exists a canonical (unique map) going from the original ring to its quotiented by \(I\)
\[\pi_I:R\to R/I\] Sending \(r\in R\) to \([r]_I:=r+I\).
Ideals of a ring contains the information about divisibility of stuff inside \(R\): multiples and divisors.
Archetypal example. Take the ring of integers \(\mathbb Z\): all of its ideals are of the form \(n\mathbb Z\): so they are \(0\mathbb Z, 1\mathbb Z, 2\mathbb Z,...,n\mathbb Z,...\). These ideals define the relation of being equivalent modulo something. Given the ideals
\[0\mathbb Z=\{0\},\quad 1\mathbb Z=\mathbb Z=\{...,\,-2,\,-1,\,0,\,1,\,2,\,3,\,...\},\quad 2\mathbb Z=\{...,\,-2,\,0,\,2,\,4\,...\},\quad...,\quad n\mathbb Z=\{...,\,-n,\,0,\,n,\,2n\,...\},\quad ...\]
We obtain quotient rings
\[\mathbb Z/0\mathbb Z\simeq \mathbb Z,\quad \mathbb Z/1\mathbb Z\simeq 0,\quad \mathbb Z/2\mathbb Z,\quad...,\quad\mathbb Z/n\mathbb Z,\quad ...\]
And we have canonical ring homomorphisms \(a\mapsto \pi_n(a)=[a]_n:=a+n\mathbb Z\) \[\pi_n:\mathbb Z\to \mathbb Z/n\mathbb Z\]
Observe. what I've defined here are the real "mod functions": \(\pi_n(a)=a\, {\rm mod}\, n\). They are ring theoretically in nature. Also notice that \(\pi_1\) is a bit special since it is a constant function, thus the only of the mod/projection morphisms to not be a morphism of unital rings\/fields: the field \(\mathbb F_1=\mathbb Z/1\mathbb Z\) with one elements does not exists.
About iterating mods. so this makes impossible to iterate the mod funtions since they are not endofucntions. Sure, you can post-compose them with the identifications \(i_n:\mathbb Z/n\mathbb Z \to \mathbb Z\) but you will obtain idempotent functions, thus trivial dynamics. The proof goes showing that \(i_n\) inverts \(\pi_n\) on the right \(\pi_n(i_n([a]_n))=\pi_n(a)=[a]_n\), we conclude that the composition \(i_n\circ \pi_n:\mathbb Z\to \mathbb Z\) is idempotent.
All of this only to say that \({\rm mod}\, 1\) is not the
fractional part in general. It is an abuse of notation and an attempt to extend a concept belonging to integers to the real numbers.
Now someone could say: but \(\mathbb R\) is a field, hence a ring, so we can mod it out by \(1\mathbb Z=\mathbb Z\) and obtain reals number mod 1. The answer is NO. This is all an abuse of notation: the integers are not an ideal of the real numbers. Real numbers are a field and fields have not "linear subspaces", using the heuristic I used at the beginning. This happens because fields have only trivial ideals... i.e. they are 1 dimensional, and their dimension 1 cannot be decomposed further...
That means that the operation of quotienting out the integers from the real is not ring theoretic but group theoretic... so it is not the modulo operation anymore. Not properly.
The additive group \(\mathbb R\) of real numbers can be quotiented by its subgroup of integers
giving as a result the circle group \(\mathbb R/\mathbb Z\simeq S^1\) and the canonical quotient surjective morphism of groups
\[\pi_\mathbb Z:\mathbb R\to \mathbb R/\mathbb Z\simeq S^1\]
Even in this case you can think of it as folding the line so that all the integers get folded over the zero... like winding the real line. An yes, the equivalence relation induced by the quotiend reminds the modulo 1 eq. relation.
\[\pi_\mathbb Z(x)=\pi_\mathbb Z(y) \quad \iff \quad y-x \in\mathbb Z\]
i.e. \(x\) and \(y\) have the same
non-integer part.
Iterating the fractional part. Again, it turns out that if you post-compose \(\pi_\mathbb Z:\mathbb R\to S^1\) to the obvious map \(i:{}S^1\to \mathbb R\) that identifies the non-integer part in the interval \([0,1)\) you have a right inverse, and again we conclude that the non-integer part \(\{x\}=(i\circ \pi_\mathbb Z)(x)\) is idempotent: not interesting dynamics.
Note. the non-integer part of \(\pi\) is still a real number, but if you look for the fractional part properly, then you have to consider \(\pi:\mathbb Q\to \mathbb Q/\mathbb Z\). But again, your problem can't arise here since \(\pi\notin \mathbb Q\).