This is a fun one. My initial thought was this: Since the distance between where you are and where you are going is infinite, and infinity plus or minus a step in any direction is still infinity, it can be said that what you call “traveling” is not moving at all relative to the destination. You appear to be in absolute stillness as seen from the perspective of the infinitely distant place.
The problem is that does not have any practical application I can think of, since we are here, not there; we perceive our activity as motion relative to points finitely close to us; and we do not have direct access to that infinite perspective. We can only imagine it.
So my second thought was to seek out some definitions: According to the standard one in the dictionary, your direction is “the line or course on which you are moving.” If you accept that, then your direction is forward when the distance between you and the distant place is “infinity minus a step/instant” and back when the direction is “infinity plus a step/instant.” I think this plays out in reality as subtracting from the distance ahead (the future) and adding to the distance behind (the past) as one travels upon the line connecting where we are (here/now) to where we are going (there/then). Paradoxically, every place in the universe that is not here/now is there/then, so once again you could say motion is just an illusion. You are always here/now, and once again in absolute stillness like a fly caught in amber.
Still not very useful, so I had a third thought: One mathematical definition of line is “a continuous state of length without breadth.” Lines can be straight or curved, a straight line being the shortest distance between two points (infinite, in your example). In two dimensions, a curve is defined as “the locii of points having a specified common property,” such as “all points equidistant from a single point” forming a circle. What then is the locii of all points on your journey if they are defined as “infinitely distant from the destination?” If we think of a straight line, there might in fact be only one point that meets that criteria? If so, isn’t it where you are right now, again in absolute stillness?
Again, the problem is that dealing with infinity almost always leads to paradox, nothing very practical. For example, the number of points on the circumference of a circle of any size can mathematically be proven to be the same as that of any other circle, i.e. “infinite.” In fact, if you line circles up concentrically, you can show by using radii that there is a one-to-one correspondence between any point on any inner circle and a point on the circumference of an outer circle. Yet any child can tell you it takes a longer line to draw a bigger circle. The math of infinity often defies logic.
Have you come up with a different answer? If so, I’d enjoy reading it.