it can be a number of things I use to own 2 servers and admin 9
so here is the list
1 yours or there's NAT type might not be open
to test your bat go check it if it's not open work to get it open if yours is on open and you can't join it might be them
2 sometimes just hard resting your Xbox will get you in . unplug your Xbox wait a few min plug it back in and that works
3 . make sure your in ark to join a server . your ark needs to be in the screen with the BIG A on it then go to the host gt and join
5 settings on server host gt might change . and he may not added you as a friend back
that's most of the common problems I have encountered