Skip to content

Extend Customization Support for Bot.base_(file_)url#4632

Merged
Bibo-Joshi merged 8 commits intomasterfrom
feature/test-env
Jan 23, 2025
Merged

Extend Customization Support for Bot.base_(file_)url#4632
Bibo-Joshi merged 8 commits intomasterfrom
feature/test-env

Conversation

@Bibo-Joshi
Copy link
Member

@Bibo-Joshi Bibo-Joshi commented Jan 1, 2025 •
edited
Loading

Closes #3355

Given the spiked request for #3355 (see the graphic that I attached yesterday), I decided to go ahead and PR for this.
I decided agains the test_env flag and for customizing base_(file_)url because I think this will be easier to maintain.
For this, I implemented support for the following options:

# backwards compatibility is ensured
Bot(token, base_url="my.tld/bot")`

# string formatting
Bot(token, base_url="my.tld/bot{token}")

# pass a callable
Bot(token, base_url=lambda t: f"my.tld/bot{t}")

PS: I confirmed that the file url is https://api.telegram.org/file/bot<token>/test/… though with this approach PTB doesn't hard-code that.

@Bibo-Joshi Bibo-Joshi added the 🔌 enhancement pr description: enhancement label Jan 1, 2025
@codecov
Copy link

codecov bot commented Jan 1, 2025 •
edited
Loading

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
6582 1 6581 485
View the top 1 failed tests by shortest run time 
tests.ext.test_application.TestApplication::test_stop_running[polling]
Stack Traces | 0.791s run time 
self = <tests.ext.test_application.TestApplication object at 0x7f52e92b0e10>
one_time_bot = PytestExtBot[token=5518645411:AAGw0qhK7e4Gnj1Z2csPAC7ZakoMk56EJfk]
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f52e92b6cf0>
method = 'polling'

    @pytest.mark.parametrize("method", ["polling", "webhook"])
    def test_stop_running(self, one_time_bot, monkeypatch, method):
        # asyncio.Event() seems to be hard to use across different threads (awaiting in main
        # thread, setting in another thread), so we use threading.Event() instead.
        # This requires the use of run_in_executor, but that's fine.
        put_update_event = threading.Event()
        callback_done_event = threading.Event()
        called_stop_running = threading.Event()
        assertions = {}
    
        async def post_init(app):
            # Simply calling app.update_queue.put_nowait(method) in the thread_target doesn't work
            # for some reason (probably threading magic), so we use an event from the thread_target
            # to put the update into the queue in the main thread.
            async def task(app):
                await asyncio.get_running_loop().run_in_executor(None, put_update_event.wait)
                await app.update_queue.put(method)
    
            app.create_task(task(app))
    
        app = (
            ApplicationBuilder()
            .application_class(PytestApplication)
            .updater(PytestUpdater(one_time_bot, asyncio.Queue()))
            .post_init(post_init)
            .build()
        )
        monkeypatch.setattr(app.bot, "get_updates", empty_get_updates)
    
        events = []
        monkeypatch.setattr(
            app.updater,
            "stop",
            call_after(app.updater.stop, lambda _: events.append("updater.stop")),
        )
        monkeypatch.setattr(
            app,
            "stop",
            call_after(app.stop, lambda _: events.append("app.stop")),
        )
        monkeypatch.setattr(
            app,
            "shutdown",
            call_after(app.shutdown, lambda _: events.append("app.shutdown")),
        )
        monkeypatch.setattr(app.bot, "set_webhook", return_true)
        monkeypatch.setattr(app.bot, "delete_webhook", return_true)
    
        def thread_target():
            waited = 0
            while not app.running:
                time.sleep(0.05)
                waited += 0.05
                if waited > 5:
                    pytest.fail("App apparently won't start")
    
            time.sleep(0.1)
            assertions["called_stop_running_not_set"] = not called_stop_running.is_set()
    
            put_update_event.set()
            time.sleep(0.1)
    
            assertions["called_stop_running_set"] = called_stop_running.is_set()
    
            # App should have entered `stop` now but not finished it yet because the callback
            # is still running
            assertions["updater.stop_event"] = events == ["updater.stop"]
            assertions["app.running_False"] = not app.running
    
            callback_done_event.set()
            time.sleep(0.1)
    
            # Now that the update is fully handled, we expect the full shutdown
            assertions["events"] = events == ["updater.stop", "app.stop", "app.shutdown"]
    
        async def callback(update, context):
            context.application.stop_running()
            called_stop_running.set()
            await asyncio.get_running_loop().run_in_executor(None, callback_done_event.wait)
    
        app.add_handler(TypeHandler(object, callback))
    
        thread = Thread(target=thread_target)
        thread.start()
    
        if method == "polling":
            app.run_polling(close_loop=False, drop_pending_updates=True)
        else:
            ip = "127.0.0.1"
            port = randrange(1024, 49152)
    
            app.run_webhook(
                ip_address=ip,
                port=port,
                url_path="TOKEN",
                drop_pending_updates=False,
                close_loop=False,
            )
    
        thread.join()
    
        assert len(assertions) == 5
        for key, value in assertions.items():
>           assert value, f"assertion '{key}' failed!"
E           AssertionError: assertion 'called_stop_running_set' failed!
E           assert False

tests/ext/test_application.py:2547: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

@SupremeManatee
Copy link

SupremeManatee commented Jan 14, 2025 •
edited
Loading

Hi @Bibo-Joshi Im not sure how I should use it. I see that the latest merge is Verified and pass so It should work.
I created a test account on telegram and a new bot
This is a snippet from my code:

application = ApplicationBuilder()
.base_url("https://api.telegram.org/bot2222136402:AFFvRRxGGiTT2QQL33_44X55m66/test")
.token("2222136402:AFFvRRxGGiTT2QQL33_44X55m66")
.build())

But Im obtaining the error InvalidToken rejected by server (I double checked the token with botfather)

Thanks for your work

EDIT: I found Bot and tried in this way:

from telegram import Bot

mbot = Bot(token="2222136402:AFFvRRxGGiTT2QQL33_44X55m6", base_url="https://api.telegram.org/bot2222136402:AFFvRRxGGiTT2QQL33_44X55m6/test/")
application = ApplicationBuilder().bot(mbot).build()

But didn't work, get the same error.

I also tried installing the test-env branch:
pip install git+https://github.com/python-telegram-bot/python-telegram-bot@feature/test-env

@Bibo-Joshi
Copy link
Member Author

Bibo-Joshi commented Jan 14, 2025

Hi.

I see that the latest merge is Verified and pass so It should work.

That only means that the changes work in the proposed branch. It does not mean that it works on PTB versions that you can install from pypi. Since the PR is not yet merged, the changes are not yet in the master branch and especially not in any release available on pypi.

I also tried installing the test-env branch:
pip install git+https://github.com/python-telegram-bot/python-telegram-bot@feature/test-env

This is what you have to do if you want to use the changes from this PR, correct.

I created a test account on telegram and a new bot
This is a snippet from my code:

application = ApplicationBuilder()
.base_url("https://api.telegram.org/bot2222136402:AFFvRRxGGiTT2QQL33_44X55m66/test")
.token("2222136402:AFFvRRxGGiTT2QQL33_44X55m66")
.build())

But Im obtaining the error InvalidToken rejected by server (I double checked the token with botfather)

The exact token must not be a part of the base url. You should pass https://api.telegram.org/bot{token}/test for the base url and https://api.telegram.org/file/bot{token}/test for the base file url. PTB will insert the token correspondingly.

harshil21
harshil21 reviewed Jan 22, 2025
View reviewed changes
Copy link
Member

@harshil21 harshil21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks pretty good

harshil21
harshil21 approved these changes Jan 22, 2025
View reviewed changes
Copy link
Member

@harshil21 harshil21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Bibo-Joshi Bibo-Joshi merged commit 5dd7b8f into master Jan 23, 2025
26 checks passed
@Bibo-Joshi Bibo-Joshi deleted the feature/test-env branch January 23, 2025 05:01
@SupremeManatee
Copy link

SupremeManatee commented Jan 23, 2025 •
edited
Loading

Hi @Bibo-Joshi, I did update the code using:
pip install --upgrade python-telegram-bot
Then I follow your instruction:

application = (ApplicationBuilder()
                   .base_url("https://api.telegram.org/bot{token}/test")
                   .base_file_url("https://api.telegram.org/file/bot{token}/test")
                   .token("2200136402:AAFvHKxEwiVz2UuLjt_hoXeBmpyfwhssFNA")
                   .build())

But im getting this error:

File "C:\Users\source\repos\TelegramYBTranscript\.venv\lib\site-packages\telegram\_bot.py", line 765, in initialize
    raise InvalidToken(f"The token `{self._token}` was rejected by the server.") from exc
telegram.error.InvalidToken: The token `2200136402:AAFvHKxEwiVz2UuLjt_hoXeBmpyfwhssFNA` was rejected by the server.
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000013AC6F6CF70>
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\proactor_events.py", line 116, in __del__
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\proactor_events.py", line 108, in close
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 751, in call_soon
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed

What can I do?

@Bibo-Joshi
Copy link
Member Author

Bibo-Joshi commented Jan 23, 2025

Please show the full traceback

@SupremeManatee
Copy link

SupremeManatee commented Jan 23, 2025

Please show the full traceback

2025-01-23 17:37:39,494 - httpx - INFO - HTTP Request: POST https://api.telegram.org/bot%7Btoken%7D/test2200136402:AAEiAa2zcJD5S4-dwTcZZP9-l1zWBy1GY3w/getMe "HTTP/1.1 404 Not Found"
Traceback (most recent call last):
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram_bot.py", line 763, in initialize
await self.get_me()
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\ext_extbot.py", line 1957, in get_me
return await super().get_me(
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram_bot.py", line 895, in get_me
result = await self._post(
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram_bot.py", line 619, in _post
return await self._do_post(
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\ext_extbot.py", line 354, in _do_post
return await super()._do_post(
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram_bot.py", line 648, in _do_post
result = await request.post(
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\request_baserequest.py", line 202, in post
result = await self._request_wrapper(
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\request_baserequest.py", line 381, in _request_wrapper
raise InvalidToken(message)
telegram.error.InvalidToken: Not Found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\source\repos\TelegramYBTranscript\test\main.py", line 512, in
main()
File "C:\Users\source\repos\TelegramYBTranscript\test\main.py", line 508, in main
application.run_polling()
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\ext_application.py", line 868, in run_polling
return self.__run(
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\ext_application.py", line 1069, in __run
loop.run_until_complete(self.initialize())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 647, in run_until_complete
return future.result()
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\ext_application.py", line 487, in initialize
await self.bot.initialize()
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram\ext_extbot.py", line 300, in initialize
await super().initialize()
File "C:\Users\source\repos\TelegramYBTranscript.venv\lib\site-packages\telegram_bot.py", line 765, in initialize
raise InvalidToken(f"The token {self._token} was rejected by the server.") from exc
telegram.error.InvalidToken: The token 2200136402:AAEiAa2zcJD5S4-dwTcZZP9-l1zWBy1GY3w was rejected by the server.
Exception ignored in: <function _ProactorBasePipeTransport.del at 0x000001E27310CF70>
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\proactor_events.py", line 116, in del
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\proactor_events.py", line 108, in close
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 751, in call_soon
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\asyncio\base_events.py", line 515, in _check_closed
RuntimeError: Event loop is closed

@Bibo-Joshi
Copy link
Member Author

Bibo-Joshi commented Jan 23, 2025

It looks like you've installed PTB from the Python package index (pipy.org). This PR is not contained in any release yet that's available there. This PR has been merged into the master branch which means that it will be part of the next release.

@SupremeManatee
Copy link

SupremeManatee commented Jan 23, 2025

It looks like you've installed PTB from the Python package index (pipy.org). This PR is not contained in any release yet that's available there. This PR has been merged into the master branch which means that it will be part of the next release.

Oh ok, so I’m gonna install it from the main branch then.
Thanks, I didn’t think about the release.
Any date on the next release?

@Bibo-Joshi
Copy link
Member Author

Bibo-Joshi commented Jan 23, 2025

no, we don't give ETAS.

@github-actions github-actions bot locked and limited conversation to collaborators Jan 31, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@harshil21 harshil21 harshil21 approved these changes

@Poolitzer Poolitzer Awaiting requested review from Poolitzer

Assignees

No one assigned

Labels

🔌 enhancement pr description: enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Native support for TGs test environment

3 participants

@Bibo-Joshi @SupremeManatee @harshil21